DOCUMENT RESUME 



ED 333 581 EA 023 097 

AUTHOR Allison, Derek J. ; Nagy, Philip 

TITLE A Study of Principal Problem Solving: An Introduction 

to the study. 

SFONS AGENCY social Sciences and Humanities Research Council of 

Canada, Ottawa (Ontario) . 
Apr 91 

31p.; Paper presented at the Annual Meeting of the 
American Educational Research Association (Chicago, 
IL, April 3-7, 1991). 

Speeches/Conference Papers (150) ~ Reports - 
Research/Technical (143) 

MF01/PC02 Plus Postage. 
•Administrative Problems; "Administrator 
Qualifications; Administrator Role; "Decision Making; 
Elementary Secondary Education; Foreign Countries; 
"Principals; "Problem Solving; Research Design; 
Sampling! "Work Experience 
"Ontario 



A study on administrative problem solving is 
introduced in this paper, which summarizes the theoretical and 
methodological bases of the study and outlines the research design 
and data collection process. Three research areas are explored: (1) 
the relationship between problem solving and decision making; (2) 
types of administrative problems; and (3) differences between 
educational administrators and nonexpert administrators in problem 
solving. The primary purpose was the identification and comparison of 
differences in problem-solving practices between novice and expert 
elementary school principals. The sampling process identified four 
experience groupings: aspirants, rookie, seasoned, and veteran. 
Thirty-two principals and 10 educational administration students from 
6 public school systems in London, Ontario, participated in the study, 
which involved interviews, small group discussion, questionnaires, 
and individual problem-solving exercises. Findings are presented in 
consecutive papers. Two tables provide participants' experience 
profiles and the data collection schedule. (34 references) (LMI) 



Reproductions supplied by EDRS rvre the best that can be made 

from the original document. 



PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 

IDENTIFIERS 
ABSTRACT 



A STUDY OP PRINCIPAL PROBLEM SOLVING: 
AN INTRODUCTION TO THE STUDY 



U.I DCMirrMEMTOr COUCATION 
Cm«« of Educational Raaaarcn and improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 
f^fht* document hat baan rap*oduca<l ai 
racawad from rna Ptnon or organization 
originating »t 
C Minor changat hava baan mada to tmpro*a 
raproduction quality 



a Potnta Of Of op*n?on» itatad in thi» docu- 
mant do not n#ca»»anty rapratant o««ial 
OERi poa«tK>n or POltCy 



'PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) " 



Derek J. Allison 
The University of Western Ontario 

Philip Nagy 

The Ontario Institute for Studies in Education 



[This study was funded by the Social Sciences Humanities Research Council,] 

[Paper prepared for a symposium presentation to the American Educational 
Research Association Annual Conference, Chicago, April 1991,] 



BEST COPY AVAILABLE 



1 

Professionals, such as architects, lawyers, surgeons, and teachers, "share 
in common the ability to connect solutions to problems . And they are assumed to 
possess a special expertise that enables them to do this" (Kennedy, 1987, p. 
133) . Administrators , particularly administrators who work with and through 
other professionals, as do educational administrators, must logically be expected 
to have- -or at least to aspire to- -expertise ot this kind. The hierarchical 
structure of organizations automatically ensures that problems will be brought to 
their attention. "Problems", Dwight Eisenhower, might just as easily have had 
blazoned on his desk, "end up here". Further, and as well illustrated by 
Mintzberg's (1973) research, successful administrators appear to be sensitive 
to--and often actively seek out- -disturbances, opportunities for improvement, 
and other problems as they go about their daily business . 

Even so, the mainstream literature or educational administration accords 
scant attention to this form of expertise , problem-solving typically being treated 
as a sub-element within the more grandly portrayed process of decision making. 
This emphasis is clearly illustrated by the index of the recently published 
Handbook of Research of Educational Administration (Boyan, 1988), where 
"Decision making" receives 30 entries for a total of 94 indexed pages, while 
"Problem solving" has only two entries indexing a mere fo'ir pages. Further, as 
exemplified by Simon's (1945) early work, by Litchfield's (1956) landmark article 
and, in educational administration specifically, by Griffiths (1959), Lipham and 
Hoeh (1974), Hoy and Miskel (1982) and the recent reviews in the Boyan 
Handbook (Estler, 1988; McNamara and Chisolm, 1988), the established 
literature has traditionally and typically concentrated on the application of 
normative models of rational choice to organizational decisions . This approach 
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elides interesting and important questions rooted in latent philosophical and 
functional differences between decision making and problem-solving and the role 
and pter-i of each in administrative work. What are the differences between 
problem -solving and decision making? When, how and in what ways do decisions 
transmute into problems and vice- versa ? What kinds of problems are 
encountered in administering schools and school systems? Do expert educational 
administrators recognize, analyze, and attempt to deal with problems differently 
from non -experts? 

The research project summarized in this series of papers was primarily 
concerned with the last of these questions, although attention was also paid to 
the kinds of problems encountered in school administration. Questions of this 
kind have not been entirely ignored by students and scholars of educational 
administration. Hemphill (1958), in particular, was an early proponent of the 
centrality of problem-solving in administration, a line of enquiry which was 
pursued in the major (but subsequently- -and strangely- -largely neglected) 
study of Administrative performance and personality by Hemphill, Griffiths and 
Frederickson (1962). More recently Leithwood and his associates (e.g. 1986, 
1989a, 1989b) have embarked on an extensive research program into the work 
and actions of educational administrators which explicitly addresses problem- 
solving processes and contexts . Our project complements the work of the 
Leithwood group in several ways and draws upon a similar literature base, but it 
was conceived and conducted independently as an extension of previous work by 
members of our research team (Nagy & Allison, 1988; Allison & Nagy, 1988; Nagy 
& Moorhead, 1990). 

This paper summarizes the theoretical and methodological foundations of the 
study, and outlines the design and the data collection process. The papers 
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which follow consider preliminary findings . We wish to stress that the project is 
still under way and our analysis is far from complete . These papers should 
therefore be regarded as accounts of work -in- progress , rather than final 
reports . 
The Literature 

The literature forming the background to this study comes from several 
sources. Among these are work on memory for complex phenomena, the solution 
of ill-structured problems, expert-novice differences, and problem solving in 
school administration . As more detailed discussion is available in earlier 
writings (Nagy & Allison, 1988; Allison & Nagy- 1989; Nagy & Moorhead, 1990; 
Nagy, 1990), we present here only a brief overview. 

Memory for complex phenomena . Researchers have posited several varieties 
of complex entities as organizing principles in attempting to understand memory 
for stories and sequences of events. Anderson, Spiro and Anderson (1978) 
describe schema theory, one version of such an organizing principle. Schemata 
act as mental structures; they incorporate general knowledge, and are more 
abstract than the particulars of a given situation. Interpretation of memory data 
involves matching elements in a specific situation to generic slots or 
placeholders . They report a study in which subjects read a restaurant story 
and a grocery-store story involving purchase of the same foods ; the expectation 
was that the subjects' internal restaurant schemata would impose structure and 
result in an increase in memory. The results confirmed their expectation, and 
support the hypothesis that complex material is stored by complex mechanisms . 

Cognitive theorists have debated the psychological status of schemata. 
Alba and Hasher (1983) argue that the evidence shows that stored memories are 
richer than the selected subset predicted by schema theory. This suggests, and 
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we concur , that it is appropriate to view schema theory as a method of imposing 
order on complexity, not necessarily involving any strong assumptions 
concerning the nature of human memory. This perspective gives the method the 
status of a portrayal technique , a heuristic device useful for imposing order on 
data. 

As a device for imposing order and examining differences, schema theory 
holds promise. For example, Schallert (1982) notes that schemata evolve; that 
is, they become more elaborate and specific with experience. This suggests that 
an examination of the details of story-lines across individuals might be used to 
highlight differences in specificity or sophistication, differences which in turn 
might be linked to experience or expertise . 

There is a difficulty with such a perspective, however how to decide 
what constitutes a more complete or more sophisticated version of a schema. For 
example, Horton and Mills (1984) reviewed the literature on human learning from 
a schemata perspective, using a levels-of -processing framework. They 
concluded that such an approach is plagued by the lack of an independent 
definition of depth of processing. Thus, a present limitation to the technique is 
reliance on subjective decisions concerning the adequacy or quality of particular 
pieces of data. In this project, we have somewhat ameliorated this difficulty by 
using a panel of expert judges to rate quality of our subjects' responses . 

Ill-structured problems. Frederiksen (1984) summarized Simon's original 
distinction between well-structured and ill-structured problems. The 
characteristics of ill-structured problems include greater complexity, less 
definite criteria for deciding if a solution has been reached, lack of complete 
information, absence of a "legal move generator", and no convenient list of 
accepted procedures . They also have higher verbal content and are more 
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context dependent. Most "real-life" problems would be classified as ill- 
structured. 

Methods for the analysis of ill-structured problems have evolved from those 
designed for well-structured ones (e.g. , logic puzzles, chess). These methods 
have tended to take the form of production rules or flowcharts . Voss and Post 
(1988) noted that the method chosen for the analysis of ill-structured problems 
reflects the theoretical concerns of the investigators . A variety of approaches 
has been developed to meet various concerns. For example, Larkin (1980), 
working with physics and algebra problems , has found that large-scale units are 
useful in the analysis of problem solving in such domairs . Voss , Greene , Post & 
Penner (1983), investigating how subjects would deal with the lack of 
productivity of the Soviet agricultural system, categorized statements as 
pertaining to goals or reasoning. Goal statements dealt with relatively global 
moves such as identification of major issues and subproblems, while reasoning 
statements dealt with the analysis within the structure of these subproblems. 
Finally, Lawrence (1988), in the context of judicial decision making, developed a 
model using elaborate if. . .then statements, in an attempt to capture a priori 
perspectives ("frames of reference"), which correspond, according to Voss and 
Post (1988), to the magistrates' courtroom schemata. 

Expert -novice differences . Glaser and Chi (1988) have provided a general 
survey of the major literature on expert-novice differences in problem solving. 
With respect to ill-structured problems, Voss et al. (1983) have provided useful 
commentary with respect to the Soviet agriculture problem. Typically, expert 
reasoning shows how a more general solution can solve a number of related 
subproblems. General characteristics of expert solutions included the following: 
experts did not articulate their highest level plans; none spoke of a general 
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solution strategy; they did not begin with a well-developed plan, but reviewed 
the problem and then developed one; general plans could be classified as either 
problem conversion (to one they could solve) or problem decomposition; experts 
tried for one general solution to solve the problem, using a lot of reasoning; 
novices tended to deal with low level subproblems and use little reasoning. In 
solution activity, only experts examined the implications of their solutions. 
Experts spent a lot of time on argumentation . 

School administration as problem solving . Two lines of research have 
examined the school principalship from a problem solving perspective . From an 
administrative perspective, Leithwood and Stager (1986) reported differences in 
principals' sorting of problems , their solution strategies , and the influences on 
their strategies and priorities . Their results showed that better principals use 
group solutions more often, and in different ways; they emphasise programs 
over buildings and relationships , and give fuller rationales . Better principals 
tackle fundamental rather than superficial problems , are more reflective and 
seek clarification more, and list more specific sources of information. Better 
principals are aware of "problem solving" as a definite activity and they enjoy it 
and are confident at it. 

Leithwood and Stager (1987) analyzed expert and nor.-expert protocols on 
school problems, finding similarities in the two groups on clear (well- 
structured) problems and differences on unclear (ill-unstructured) ones. On 
ill-structured problems , experts were about the same as they were on well- 
structured, but non-experts had more interpretation statements, fewer goals 
and solution processes . The experts thought of school and system 
consequences; non-experts thought of personal consequences. Experts 
expressed belief in the power of rational thought; non-experts didn't. Experts 
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were more specific in statements and assumptions, and cited more relevant 
anecdotes. Experts planned more; non-experts paid little attention to planning. 

Working more from a methodological perspective , the group reporting the 
present work have evolved a procedure through several studies, starting with 
methods closely related to Voss' work. The fundamental issue is whether 
differences in quality of response can be related to the nature as well as to the 
content of statements made. Nagy and Allison (1988) made some progress 
working with the case used in the present study. Unlike the Soviet agriculture 
real- world problem, their subjects were limited by the amount of information in 
the case . As a result , the system of statement categories was modified , yielding 
a system of 21 statement categories , no longer distinguishing between goals and 
reasoning. They also recognized the need to deal with relative expertise. Voss' 
approach did not focus on differences in quality of the responses of different 
experts . 

This problem was pursued in Nagy and Moorhead ( 1990) , using a different 
case study. Two conclusions are of interest. First, fine distinctions in the 
categorization of statements were abandoned in favour of a catch-all 
"reasoning". Second, faced with an inability to link intuitively-felt differences 
in quality with either differences in use of statement types or diagram shape , 
the authors resorted to a checklist of individuals consulted, information sought, 
and actions taken . On this basis , clear differences in relative expertise of 
subjects could be discerned , but the price was the absence of anything 
resembling cognitive science research on ill-structured problems . 

Allison and Nagy (1989), again using the same case as used here, examined 
the links between cognitive science views of subjects 1 responses and the more 
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traditional educational administrative and personnel assessment perspectives. 
They began with a diagrammatic portrayal method, but found it unconnected 
with the other views of the transcripts , and with any intuitive notions of what 
"felt right". They formed several conclusions: first, diagrammatic portrayals of 
problem solving in school administration may not capture the data successfully; 
second, more probing of subjects may be required to deal with "things left 
unsaid" and individual differences in speaking style; and third, something like a 
context bound "marking scheme", and not merely a system of categorizing 
statement types, is required to differentiate between response quality. The 
present study has incorporated many of these suggestions. 

Finally, Nagy (1990), working with group discussions of children dealing 
with a hypothetical family problem situation involving chores and allowances, has 
made an initial attempt to apply a schema approach to problem solving. His 
method involved a very detailed categorization of statements , followed by a 
categorization of the categories. The result was a set of thirteen statement 
types (actions and positions), which could be judged as representing minimal, 
typical, and enriched schemata of what constitutes fairness within a family. He 
concluded that individual elements or statements within discussions could be so 
classified, but lhat classification of entire discussions was rmpractical. 
However, it seems clear that schema theory, unlike diagrammatic portrayals, 
provides a mechanism for capturing enough context to make judgments of 
response quality. 
Design 

Given the theoretical context outlined above, the main objective of the 
study was to investigate and compare differences in problem solving between 
novice and expert principals . The study was delimited to the elementary school 



10 



principalship on the grounds that organizational and other differences between 
elementary and secondary schools would unnecessarily complicate matters . For 
reasons discussed later, the scope of the study was further limited to subjects 
residing within commuting distance of London, Ontario. 

Our first design problem was how best to conceptualize the novice-expert 
continuum and select representative participants for the study. Defining and 
selecting members of the "novice-principal" (later renamed Rookie) group 
seemed relatively straightforward, the sole selection criterion being that 
participants had to have been recently appointed to their first principalship . We 
were aware, however, that new principals may have gained considerable 
familiarity with the problems of school administration through long service in 
subordinate positions of added responsibility [ SPARs ] , particularly the vice- 
principalship. Some may even have gained de facto experience before becoming 
de jure principals by, for example, serving as a vice -principal of a twinned 
school. We attempted to control for the potentially confounding effects of 
experience gained in SPAR situations by including a group of Aspirant 
principals in the study. Members of this group were recruited from graduates of 
the Ministry of Education Part II principal certification course held in London in 
the summer of 1988 or 1989. Possession of this certificate and the prerequisite 
Part 1 certificate is a necessary qualification for appointment to the principalship 
in Ontario and we assumed that these graduates would be seeking such an 
appointment in the near future. Further, many Ontario school systems now 
routinely require completion of one or both of the principal's certification 
courses before appointing individuals to vice-principalships , a practice that we 
anticipated would limit the amount of SPAR experience within the Aspirant 
group. 

li 
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Selecting an operational definition for identifying and recruiting "expert" 
principals was more difficult . We considered asking superintendents , peers 
and /or teachers to nominate "expert" principals, but rejected this approach on 
the grounds of inherent reliability , validity and ethical difficulties . We also 
considered using an independent measure of principal effectiveness to select our 
experts , but were uncomfortable with the limitations and possible biases which 
could result from this approach and thought that the p^e- selection of 
participants through such a process might introduce a distorting "halo" effect 
into the study. Consequently we decided to select our "experts" solely on the 
basis of length of experience in the principalship. Experience, particularly 
experience pained in professional roles, is often associated with — indeed, is 
commonly taken as indicative cf--expe:*tise. Indeed, the two words share the 
same root; and while a novice by definition has little or no experience, an expert 
is commonly assumed to have a great deal. Further, one of the four scholarly 
definitions of expertise identified in Kennedy's (1987, p. 148) review of the 
literature is based on the view that "expertise evolves and develops with 
experience". Following Schon (1983; 1987) and Schwab (1978), she cautioned, 
however, that while experience appears to be a necessary precondition for 
expertise, time -in -role does not, by itself, necessarily produce experts, for 
"experience can only contribute to expertise if practitioners are capable of 
learning from it". 

Thus, while our decision to select "expert" principals solely on the basis of 
experience considerably simplified the identification and recruitment of subjects , 
it ran the risk of failing to "capture" truly expert problem-solvers. Expressed 
another way, we had no way of knowing in advance whether the experienced 
principals selected for the study would turn out to be any better at problem- 
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solving than our Rookie, or even Aspirant, principals. Nevertheless, we viewed 
this aspect of the design as a major strength for it would allow us to test directly 
the degree to which experience-in-role is associated with problem-solving 
expertise and technique. To further strengthen this element of the design we 
decided to recruit one group of moderately experienced and another of highly 
experienced principals . In selecting the first of these groups , which we termed 
"Seasoned", we sought principals with 10-15 years of experience. The highly 
experienced group, which we dubbed our "Veterans", were selected on the basis 
of 20 or more years of experience gained in at 'east three different schools. 
Thus the four experience groupings originally defined in this study were 
Aspirants (qualified but not experienced" , Rookie (in the first one or two yearr 
of experience), Seasoned (10-15 years of experience) and Veteran (more than 20 
years) . 

Experience thus became the initial dependent variable in the design , with 
problem-solving technique and expertise being assessed through post-hoc 
analyses of how study participants "thought-through" a standardized case 
problem. In order to expand the scope of the study and increase the 
opportunities for triangulation within the data we also decided to conduct 
extensive interviews with study participants and gather data from a battery of 
pencil-and-paper instruments. Taken together, these activities would clearly 
require a substantial time commitment from participants. Consequently we 
decided to collect all of our data through a series of day-long research sessions 
held at the University of Western Ontario. In addition to standardizing the data 
collection process and minimizing the possibility of interruptions , this approach 
also had the desirable effect of removing the study participants from their 
working environment, a design feature which would, we hoped, better focus 
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their attention on the tasks we wanted them to perform. At the same time, 
centralizing data collection had the less desirable result of limiting potential 
participants to principals and aspirant principals residing within reasonable 
commuting distance. 
R ecruitment 

Given this limitation, we recruited all participants from six public school 
systems within the immediate vicinity of London, Ontario. Proximity was not the 
sole selection criterion , for we wanted to draw principals from both rural and 
urban contexts . We also elected to exclude Roman Catholic systems in order to 
control for this variable. Formal permission to recruit participants was sought 
from and granted by the boards of each of the six systems selected. Letters 
were then sent to all elementary principals and graduates of the Principal's 
certification course in thone systems explaining the study and inviting them to 
participate . Reply forms were included with the letters mailed to principals 
asking respondents to indicate the year in which they were first appointed as 
principals and the number of schools in which they had served as principals . 

Approximately sixty principals and fifteen graduates of the principal's 
certification course volunteered to participate. Approximately forty-five of the 
principals satisfied our pre-determined criteria for inclusion in one of the 
relevant three experience categories . We planned to recruit a total of forty 
participants distributed equally across the four experience categories . We 
further planned to collect data from four participants at once, which « >uld 
require a total of ten data collection sessions, each lasting a single day. 
Appropriate dates were selected and our volunteers were then contacted by 
telephone to schedule attendance at the data collection site. The scheduling task 
proved to be quite difficult, with a number of volunteers being unavailable on 
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some of our predetermined dates . Unforeseen circumstances also conspired to 
eliminate two of our scheduled data collection days , reducing the total number of 
possible participants to 32, although we were successful in recruiting an equal 
number of participants in each of our four experience categories. 

After the data from the original four groups of principals had been 
analyzed , and it became evident that experience and expertise do not seem to be 
highly correlated, it was felt that there would be merit in adding, for the sake of 
comparison , a category of subjects who could truly be considered complete 
novices , in that they had not only had no experience in school administration , 
but had not yet begun their teaching careers . It can be argued that 
socialization into the administrative culture of schools begins with teachers k and 
thus that there are no true novices within the professional ranks . The datapool 
was, therefore, augmented by the addition of ten pre-service student teachers, 
and this group was dubbed Entrants. In the fall of the second year, pre-service 
students attending the Faculty of Education of The University of Western 
Ontario were invited to participate in the project. Volunteers were scheduled 
for data collection at mutually appropriate times , and all were handled 
individually. As will be explained later, these students weve only asked to 
complete two portions of the data collection. 

Table 1 summarizes years of relevant experience held by study participants 
in each of the predetermined experience categories . The Table shows that , as 
intended , none of the members of the Aspirant group had had experience as a 
principal, and on average they had gained less than one year of experience in 
subordinate positions of added responsibility. Indeed, the maximum SPAR 
experience in this group was one year. Most of the Rookies, on the other hand, 
had considerable SPAR experience, one member of this group having amassed 22 
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years of experience in such positions . All of the Rookies , nevertheless , had 
only been principals for three years or less , four of the eight having been 
principals for only one year or less . None of participants in our two experienced 
groups had had less than ten years of experience as a principal, our most 
experienced Veteran having served as principal for 28 years in 8 different 
schools . Table 1 also shows a clear break between experience in the 
principalship for the Seasoned and Veteran groups, members of the Seasoned 
group having been principals for an average of 13.6 years (minimum = 10 yrs; 
maximum = 16 yrs) and members of the Veteran group for an average of 23.9 
years . While our participants were widely distributed in terms of administrative 
experience , Table 1 also shows that there was much less variation in the total 
number of years spent working in schools than might be expected, members of 
the Rookie group having, on average, begun their educational careers only a few 
years after members of the Seasoned group, and only five years or so after the 
Veterans . This was an unexpected but nonetheless intriguing development 
which is discussed more fully in one of the accompanying papers . 
The data collection process 

Each of the original round of data collection days (i.e. for the Aspirant, 
Rookie, Seasoned and Veteran groups) followed the same pattern, as summarized 
in Table 2. Participants were deliberately grouped together by experience, so 
that four members from the a single experience category attended each data 
collection session. Grouping participants together in this way, we reasoned, 
would minimize differences in status and help create a more comfortable and less 
threatening social setting. Participants were met on arrival at the research site 
and invited to a comfortable room for coffee and conversation with the 
researchers. When all of the four participants scheduled for a given day had 
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arrived , they and the four members of the research team assembled in a seminar 
room where the principal investigators presented an overview of the research 
project, explained the activities planned for the day, and invited questions from 
participants . 

Once participant questions had been answered the principal investigators 
initiated a 10-20 minute round-table discussion on the topic of problems 
encountered in school administration . This activity was intended to serve four 
purposes. First, we thought that the sharing of experiences and opinions within 
a supportive small-group setting would further "break the ice" and help 
participants feel more at ease with the researchers, each other, and the research 
topic . Second , the discussions served to focus the attention of participants on 
the subject under investigation . Third , with the permission of the participants 
(none refused) these discussions were tape-recorded, thus allowing participants 
to become familiar with this aspect of the research procedure. Fourth, we 
thought that the content of the discussions might well yield useful insights into 
the problems encountered in school administration and the ways in which 
principals (and aspirants) think about them. Whether this will turn out to be 
the case is unclear at this stage as the tapes have not yet been analyzed. 

After the group discussion, each participant was paired with a member of 

the research team, and the four participant-researcher pairs adjourned to 

separate rooms for the fact-finding and think-aloud exercises. The researchers 

first obtained permission to tape-record the session and then read the following 

instructions to participants : 

In this hour I will be asking you to take on the role of a principal 
recently appointed to a new school, about which you know very little. 
There are two parts to this exercise: first there is a fact-finding part, 
to give you the opportunity to obtain additional information about the 
school by asking me questions; secondly, you will be given a detailed 
report on a specific problem in the school and asked to think aloud as 
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you read the case and think about how you would respond . We will do 
the fact finding phase first. Immediately after we finish this phase, I 
will give you the report on the specific problem in the school. The 
purpose of this fact finding session is to give you a chance to ask 
questions about the school, its organization, its staff, its context. 
You don't need to worry about the specific problem yet; this is an 
opportunity to learn about the school . 

You can start asking me questions now , and please tell me when you 
feel you are ready to stop asking questions and go on to the case . 

Some participants responded by seeking clarification of the task. The 
researchers answered such questions by paraphrasing the instructions given 
above. When participants asked questions about the problem which was to be 
presented, the researchers replied that the nature of the problem would become 
clear from the written description to be presented latter. 

The purpose of the fact-finding exercise was to generate data about the 
background and contextual information sought by participants in preparation for 
the problem-solving task which they knew lay ahead. In order for the exercise 
to work well and not distort the problem-solving activity which was to follow, we 
had to be able to provide plausible and c nsistent answers to likely questions, 
answers which would accurately mesh with and complement the problem which 
was to be presented. We prepared for this by developing a detailed profile of 
the fictitious school which provided the setting for the case problem. Some of 
this information, such as total enrolment and the location and age of the school, 
was included in the case study which presented the problem. The bulk of our 
profile information, however, including staff size and composition, grade 
enrolment, instructional program, community characteristics and school system 
organization and policy, was not included in the case and was only made 
available to participants if they asked appropriate questions during the fact- 
finding session. To ensure plausibility we asked three experienced 
superintendents to examine our initial profile and point out any oddities and 
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inconsistencies and a number of adjustments were made in the light of their 
comments . Members of the research team then memorized the profile information 
prior to the first fact-finding session. To help ensure consistency, each 
researcher also had access to a copy of the profile during the fact-finding 
sessions as an aide-memoir . Inevitably some of the participants asked questions 
for which we had no pre-determined answers. One, for example, asked how the 
staff room was furnished; another asked about student washroom facilities. In 
such circumstances , that is when questions were asked for which we had no 
prepared answers , the researcher simply replied that he or she had no 
information on that point. 

All of the participants used the fact-finding session to seek background 
information about the problem, but some sought considerably more information 
than others . The highest number of questions asked by a participant was 85 ; 
the lowest six . Preliminary results from our analysis of the transcripts of the 
fact-finding activity are presented in one of the accompanying papers . 

When a participant declared that he or she did not wish to ask any further 
questions , the researcher introduced the think-aloud problem analysis exercise 
by reading a set of prepared instructions. These instructions asked 
participants to first read aloud the case which the researcher would provide; 
verbalizing any thoughts that occurred as they did so, and then think aloud as 
they worked on the problem presented in the case . Participants were 
specifically instructed to try and verbalize all of their thoughts from the moment 
they were handed the case until they had finished outlining their proposed 
course of action for dealing with the problem preserved. Before embarking on 
the case problem itself, a short training session wai held in which participants 
were asked to think aloud while working on two practice problems , both of which 
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were presented in written form so as to better conform to the requirements of the 
main task. After the first practice problem ("How many windows are there in 
your house?") the researcher provided corrective feedback and encouragement 
in order to help the participant become more proficient and comfortable with the 
think-aloud process. The second trial ("You are on your way to work and your 
car won't start") was provided for further practice. Once both trials had been 
completed, participants were handed a copy of the "Miss MacDonald" case, which 
i£ included in an appendix to the accompanying Nagy paper, and asked to begin 
reading and thinking aloud . 

Verbatim transcripts have been prepared for the fact-finding and think- 
aloud sessions, but equipment malfunctions resulted in the loss of three sessions 
(1 in the Seasoned group and 2 in the Veteran group) . Certain aspects of these 
lost sessions have been reconstructed from the interviewer's notes , but these 
were not considered sufficiently complete for full analysis purposes . The 
preliminary results of the fact-finding and think-aloud sessions presented in 
accompanying papers are thus based on a total of 29 transcripts . The paper by 
Nagy offers a detailed description of the method used to analyze the transcripts 
and emergent findings . As explained in more detail in the first of the Allison & 
Allison papers , three external judges have also rated the relative quality of the 
actions participants proposed to take in dealing with the Miss MacDonald 
problem. A high level of agreement was evident between the judges' ratings and 
they were subsequently collapsed into a single mean quality score which was 
used as a differentiating variable in our preliminary analyses . 

The student teachers who were included in the second round of data 
collection were at this point dismissed with our thanks, since the subsequent 
data collection activities were considered to be irrelevant to them. In the 
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original round of data collection, however, when the think-aloud task was 
completed, participants and researchers gathered for a refreshment break, after 
which the participants completed a battery of pencil and paper tests . The 
battery of instruments included an extensive personal profile questionnaire 
designed to collect information about the interests , skills , and educational and 
career histories of participants. In addition, participants were asked to 
complete the Ghiselli (1971) Self- Description Inventory , a well- regarded 
inventory of managerial ability (Ghiselli, 1971), Fiedler's (1967) LPC Scale , a 
somewhat enigmatic but nonetheless widely used measure of leadership style, a 
role description instrument for the principalship originally used in the National 
Principalship Study (Gross & Trask, 1976), and the "worry scale" employed in 
the same survey, which seeks to identify aspects and elements of school 
administration which are of particular concern to respondents. Two instruments 
developed by the research team, a problem- solving orientation questionnaire and 
a values scale, were also included in the battery. 

Most of these instruments were selected or developed to yield data for 
testing plausible hypotheses concerning problem-solving style and expertise. 
We were interested in investigating, for example, relationships between problem- 
solving expertise, experience, and scores on the Ghiselli scales, particularly the 
Initiative , Decisiveness , Self -Assurance and Supervisory Ability measures . 
Similarly, we anticipated that there would be a discernible relationship between 
value orientations as measured on our values scale and problem-solving styles 
identified through analysis of the fact-finding and think-aloud transcripts . 
Other instruments were included to increase the opportunities for triangulation 
within the data. Responses to the problem- solving orientation questionnaire, 
for example , should allow us to compare self -declared stances toward problem- 
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solving with the actual patterns of thinking revealed in the case analysis 
exercise. Finally, other instruments were primarily intended to help clarify and 
control for differences between participants in our four experience categories. 
The personal profile questionnaire, for example, provided data on the 
education, including graduate degrees, of participants. 

All of the raw data from the instrument battery has been coded and entered 
into data bases and analysis is in progress. Early results from the analysis of 
selected instruments are presented in the first of the Allison & Allison papers. 

Lunch followed the instrument battery, after which each participant was 
interviewed by the same researcher who had administered the morning fact- 
finding and think-aloud session. All of the interviews were recorded, and later 
transcribed, and each researcher also made extensive written notes. Both the 
notes and transcripts of the recordings were used for analysis. The interview 
protocol was divided into two parts . The first section contained six main 
questions and five follow-up questions or probes which were intended to obtain 
evaluative and reflective comments on the fact-finding and think-aloud 
activities. The first question, for example, asked respondents how realistic the 
Miss MacDonald case was in their view. Other questions sought to identify how 
participants had defined the problem and what they saw as the main variables 
and important issues. There was also a check question which asked participants 
how informed they considered themselves to be about school libraries and how 
knowledgeable they were about the Partners in Action document referred to in 
the case. 

To end the first part of the interview the researcher showed the participant 
a list of nine major and six subsidiary factors relevant to the Miss MacDonald 
problem. Participants were then asked if there were any items on the list that 
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they had thought of but not mentioned during the think-aloud exercise or if 
there was anything on the list that they did not consider at that time but in 
retrospect thought might be relevant. These questions were used as a means of 
encouraging participants to reflect on and review how they had analyzed the 
case , with the researchers prompting participants to consider each of the factors 
in turn and probing for explanations of any major omissions or assumptions 
which had not been evident . 

The second part of the interview dealt in more general terms with the 
problems encountered in school administration and how, in the perceptions and 
experience of our participants , they are best handled . A modified but largely 
parallel version was used for members of the Aspirant group who, by definition, 
had no direct experience of the principal's role. The interview protocol for 
principals included 27 main questions as well as a number of subsidiary probes . 
Examples include a question which asked respondents to describe a specific 
problem they had dealt with recently and how they had handled it, another 
which asked them to identify and describe any "rules of thumb" they used in 
handling problems and another which asked what advice they would give to a 
new principal about problem -solving. 

The full interview, that is both parts, took between 1^-3 hours depending 
on the loquacity of the interviewee and /or the tenacity of the interviewer. 
Analysis of the notes and tapes has begun, but much yet remains to be done. 
Some preliminary results are presented in the second of the Allison & Allison 
papers. 

Conclusion 

The detailed overview of the design and execution of the study offered in 
this paper is intended to serve as both an introduction to the accompanying 
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papers and as a "road map" to help understand how the component parts of the 
project fit together and are related to each other. Each of the accompanying 
papers summarizes and comments on the preliminary results emerging from one of 
the main components of data collection process, the sequencing of the papers 
following the order in which each component was described in this paper. Thus, 
the first of the accompanying papers (Moorhead) focuses primarily on the fact- 
finding data, the Nagy paper which follows deals with the analysis of the think - 
aloud transcripts , the third paper discusses the quantitative analysis of the 
pencil and paper and related data and the fourth and final paper offers a 
qualitative analysis of the interview findings. 

This study is by no means complete. Because there was so much data 
generated , large portions remain unanalyzed , and an examination of the inter- 
relationships between the various data sections has hardly begun . We intend to 
obtain quality assessments of the think-aloud transcripts from a variety of 
judges , partly to further refine our analysis of the original subjects and partly 
to provide a second level of data collection by recording and analyzing their 
discussions of the responses of our subjects . We would also like to extend our 
data base to include a category of experienced managers and administrators from 
outside of education — subjects with expertise in a different context. 

Finally, one other reason for providing a detailed account of the data 
collection procedure used in this study is that we thought the process itself 
might be of some general interest. On first encounter the process may appear 
somewhat formidable, but we were very pleased with the way it worked in 
practice and we believe our participants found the experience enjoyable, 
stimulating and worthwhile. It was also, we feel, an economical, efficient and 
controlled way of collecting a large body of varied data . Whether the game was 
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worth the candle will of course depend on what we finally learn from these data. 
But even if the final results prove disappointing, we believe that the data 
collection process which was developed for this study is in itself a valuable 
invention. 
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TABLE 1 



EXPERIENCE PROFILES OF STUDY PARTICIPANTS BY GROUP 



1 


CLTCHX 


SPARX 


PRINX 


TOTSCHX 


TOTPRIN 




N 


Mean years of experience 




Mean 


Entrants 


10 


0 


0 


0 


0 


0 


Aspirants 


8 


15.6 


0.9 


0.0 


16.5 


0.0 


Rookies 


8 


11.8 


9.4 


1.6 


22.6 


1.5 I 


Seasoned 


8 


8.1 


3.6 


13.6 


25.4 


3.6 I 


Veterans 


8 


3.9 


0.4 


23.9 


28.1 


5.0 1 


TOTAL 


42 


9.8* 


3.6 


9.8 


23.2 


2.5 | 



CLTCHX = Years of experience as a classroom teacher only 
SPARX = Years of experience in subordinate positions of added 

responsibility 
PRINX = Years of experience as a principal 
TOTSCHX = Total years of professional experience 

(CLTCHX+SPARX+PRINX) 
TOTPRIN = Number of principalships held (i.e. in different schools) 

* Means in this line are for the Aspirant , Rookie , Seasoned and Veteran 
respondents (N~32), 
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TABLE 2 
DATA COLLECTION SCHEDULE* 
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8:30 
9:00 
9:45 

10:45 
11:00 

12:15 
1:30 



Participants arrive, meet researchers, have 
informal coffee. 

Full group session: introductions, discussion of 
typical problems faced by principals. 

Participants and researchers pair off and 
complete the fact-finding and the think-aloud 
exercise . 

Coffee . 

Participants complete pencil and paper 
instruments. 

Lunch, hosted by researchers. 

Participant /research pairs complete the interview 
schedule (no time limits). 



For Aspirant, Rookie, Seasoned and Veteran groups only. 
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